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Summary. We formulate two types of extensions of the large deviation theory initiated by 
Bahadur in a non-regular setting. One can be regarded as a bound of the point estimation, the 

^ ■ other can be regarded as the limit of a bound of the interval estimation. Both coincide in the 

regular case, but do not necessarily coincide in a non-regular case. Using the limits of relative 
Renyi entropies, we derive their upper bounds and give a necessary and sufficient condition 
for the coincidence of the two upper bounds. We also discuss the attainability of these two 

Ph ' bounds in several non-regular location shift families. 
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1. Introduction 
(N 

^ . As is known, Fisher information gives the bound of the accuracy of estimation. However, 

this fact holds only for a regular distribution family. Hence, if the distribution family 
does not satisfy the regurality condition, we have to treat the information quantity as 
t::;^- ' alternative of Fisher information. In this paper, we consider this problem only for the 

^D , location shift family on the real line M. such that the support depends on the true parameter. 

^5 ' So far, this problem has been mainly studied concerning the mean square error. This paper 

treats this problem with the large deviation criteri on, which was introduced by Bahadur 



1-^ ' (JBahadur (1960)1 : iBahadur (1967)1 : iBahadur (1971)1 ). Owing this method, we can discuss 



j^ , the difference between the interval estimation and the point estimation. For this analysis, 

Ch ' we introduce two type extensions of Bahadur's large deviation bound. One is the quantity 

• • , cn(^), which can be regarded as the limit of the accuracy of the interval estimation. The 

j^ ' other is the quantity a2{0), which can be regarded as the limit of the accuracy of the point 

estimation. We also show that these two quantities coincide in a regular distribution family, 
^ ' but they do not coincide in a non-regular distribution family. 

In order to evaluate the actuary of a sequence of estimators T — {T„} for a probability 
distribution family {pe\6 G 0} from the large deviation viewpoint, Bahadur focused on 
the error probability Pg{\Tn — 0\ > e}, which goes to exponentially. That is, the rate 
of the quantity — logpg{|T„ — 9\ > e} is in order n. Hence, he discus sed the following 



exponential decreasing rate of the error probability with a fixed error bar (JBahadur (I960) 



Bahadur (1967)1 : IBahadur (1971)h : 



/3(f,0,e):=liminf — logp^{|T„-0| > e}. (1) 

n 

For this purpose, he treated KuUback-Leibler's relative entropy (divergence) D{p\\q) := 
J p(lu) {log p{uj) — log(3'(w)) (ko. Using the relation between its limit and Fisher information, 

he characterized the slope a(r, Q) := lim^^o ^^ ''^ of the exponential rate by the Fisher 
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information. Since this discussion is a fundamental for this paper, it is summarized in 
Section [2] 

However, this method cannot be apphed to the family in which the support depends on 
the true parameter, because KuUback-Leibler's relative entropy diverges in this case. On 
the other hand, it is not impossible to define Fisher information even if Kullback-Leibler's 
relative entropy diverges. In this paper, we focus on the the limit of the relative Renyi 
entropies I''{p\\q) := —\o^fp''(ijj)q^~''(ij)d u (0 < s < 1), which is deeply treated in a 
non- regular location shift family by Hayashi ( Havashi f2002)b| ). In Section [31 we define two 



quantities ai{9) and 0:2(0), and give their upper bounds ai{d) and 02(0). Both of ai{9) 
and 02(0) are defined as the optimal slope of the exponential rate /3(T, 9, e) concerning e at 
the limit e — > 0, but their definitions are slightly different. Also we derive a necessary and 
sufficient condition for the coincidence of these two upper bounds. Since these upper bounds 
are proved by a very general treatment in Section [H they are valid for general non-regular 
families. In Section [51 we focus on several estimators useful for location shift families. We 
calculate their exponential rates /3(T, 0, e) and their slopes a{T, 9) concerning e at the limit 

2. Bahadur theory 



In this section, we begin by summ arizing the results reported by Bahadur ( Bahadur (1960)t 



Bahadur (1967)1 : [Bahadur (1971)h . who discussed the decreasing rate of the tail probability 
in the estimation for a distribution family. Given 71-i.i.d. data wi, . . . ,a;„, the exponential 
rate /3(T, 9, e) of the estimator T = {T„} is written as 

/3(f, 9, e) = min{/3+(f , 0, e), ^ (f , 0, e)}, 

where the exponential rates of half-side error probabilities are given by 

/3+(f,f?,e):=liminf — logp^{T„ > 9 + e} 
n 

p-{f,9, e) :- liminf — \ogp'^e{T^ <9-e}. 
n 

When an estimator T = {Tn} satisfies the weak consistency 

P'SUTn-Ol >e}^0 Ve >0, V0 G 6, 
using the monotonicity of KL-divergence, we can prove the inequality 

/3(f, 9, e) < min{Dipg+,\\pg),Dipe-,\\pe)}. (2) 

Note that if, and only if, the family is exponential, there exists an estimator attaining the 
equality ([2]) at V6' g 0, Ve > 0. Therefore, for a general family, it is difficult to optimize the 
exponential rate /3(T, 0, e). 

Instead of the exponential rate f3{T, 9, e), We usually consider the slope of the exponen- 
tial rate: 

a{f,0):= lim l/3(f,^,e). (3) 
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In this case, when the Fisher information Jg satisfies the condition 



Je := / lg{ujYpg{duj) = hm —D{pg+^\\pg), lg{u)) := — -logpe(w), (4) 

e-»o e^ at) 



the inequality 



a[f,e)<]^Je (5) 



holds. Moreover, as was proven by Fu (jFu f 1973)1 ). if the family satisfies the concavity of 



the logarithmic derivative lg{u}) for 9 and some other conditions, the MLE 9ml attains the 
equality of ([5]). These facts are summarized in the two equations: 

a{9) := sup a{f,9) = ]-Jg (6) 

f:WC 

lim 4 sup /?(f,0,6) = ije. (7) 

As is mentioned later, these equations imply an interesting relation between the point 
estimation and the interval estimation. 



3. Upper bounds 

In this paper, the relative Renyi entropies P(pe||pe+e) substitute for the KL divergence. 
Note that the order of ^{pgWpg+f) is not necessarily e^ at the limit e ^ 0. However, its 
order is independent of the parameter s, as is guaranteed by the inequalities 

2min{s,l - s}I^ {pg\\pg+^) < I%pg\\pg+^) < 2 max{s, 1 ~ s}I^pg\\pg+^), 

which are proven in Lemma[n]of the Appendix. In several cases, the order of the exponential 
rate (3{T,9,e) coincides with the order of the relative Renyi entropies I''{pe\\pe+e)- In the 
following, we use a strictly monotonically decreasing function g{x) such that P{pg\\pg+f_) = 
0{g{e)) and g(0) = 0. 

Following equations ([6]) and ([7]), we define two extensions of Bahadur's bound (slope) 
a{9) as 

ai(6') :=limsup— -sup inf P(f,9',e) (8) 

e^+o .g(e) f e-e<e'<e+e ^ ' 

a2{9):=SMpa2if,9), (9) 

f 

where 

a^if, 9) := liminf ^T , jnf ,^ /3(r, 9', e). (10) 

In the interval estimation, we consider only the error probability concerning the fixed in- 
terval [9 — c,9 + e]. That is, the optimization depending on the quantity e is allowed. 
Hence, the performance with enough small e is characterized by ai{9) not by a2{9). On 
the other hand, in the point estimation, we have to treat the error probability concerning 
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the interval [9 — e,9 + e] for all e. The performance of an estimator T is characterized as 
the limit a2{T, 9). Then, the bound of the performance of the point estimation is given by 
Q^2(^) '■= supj5 Q!2(r, 0). In the regular family, equation ^ can be regarded as the bound 
of the point estimation, while equation ([7]) can be regarded as the limit of the bound of 
the interval estimation because sup^;.^^ /3(r, 0, e) corresponds to the bound of the interval 
estimation with width 2e of the confidence interval. Therefore, we can conclude that there 
is no difference between the point estimation and the limit of the interval estimation in 
the estimation in the regular family. In the following, we consider whether there exists a 
difference between them. 

Note that we take infimmir inf3_p<fl/<fl _i_f into account in (|10p . unlike ([3]). As was 



pointed out by Ibragimov and Has'minskii (jibragimov et al. fl981)l ). when KL-divergence 






is infinite, there exists a consistent super efficient estimator T such that /3(T, 9, e) and 
limg^+o —f^P{T, 9, e) is infinite at one point 9. Therefore, we need to take the infimum 
inf0-c<0'<0+c into account. In this situation, we do not need to limit estimators to weakly 
consistent ones. As is proven in the next section, we can obtain the following theorems. 

Theorem 1. When the convergence lim^^o — (e) ^'^'^ ** uniform for < s < 1, 
the inequality 

ai{e) <ai[9) -.^ 2^ sup /„% (11) 

0<s<l ' 



holds, where k and H g are defined by 



I''{P0-e/2\\Pe+e/2) i^g^Q 



,^ -/-— ^-^-- 1 > s > (12) 

e^+O g{e) 

^- = lini ^. (13) 

.^+0 g{e) 

Lemma[^ proven in Appendix\^ guarantees the existence of such a real number k. 

Note that the function s — > H is concave and continuous because the function s — > 
I'^{Pe-ie\\Pe+^e) is concave and continuous. Therefore, when Ig g = l],~g^ ^ we have 

ai{e)^2^A. (14) 



Theorem 2. // the convergence lime_^o — ^/2JPa+^/2) ^^ uniform for s G (0,1) and 
9 G K for any compact set K cR, the inequality 

{suPo<.<ii(fe)(s^ + (l--5)^) tfK<l 

2/|e */k=1 (15) 

holds. 
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In our proofs of these theorems, Chernoff's formula and Hoeffding's formula in simple 
hypothesis testing play importa nt roles. 



As was proven by Akahira (jAkahira (1996)1 ) . under some regularity conditions for a 
distribution family, the equation 

holds. When we choose g{x) — x'^, we have k — 2,1^ g = ^Jgs{l — s), and the relations 

ai{9) = 4 max ^g^ -Jo 

Q<s<l "■ 2 

f^ 1 

a2(0) = mm — == -Jg 

o<s<i s(l - s) 2 

hold. In particular, if the distribution family satisfies the concavity of the logarithmic 
derivative lg{oj) for 9 and some other conditions, the bound ^Jg is attained by the MLE. 
Thus, the relations ai{6) — 02(6*) — ai{9) — a2{0) — \Je hold. 

As a relation between two bounds Si (6') and a2{9), we can prove the following theorem. 

Theorem 3. The inequality 

ai{e)>a2{B) (17) 

holds, and |J7[ ) holds as an equality if and only if the equations 

aM = '^''lie (18) 

2%^,=S2(0) (19) 

hold. When n <1, ([J7[) holds as an equality if and only if equation \1^) holds. 



When P g is differentiable, condition P^ is equivalent to -^/^ 5 



= 0. 



4. Proofs of upper bounds 

In our proofs of Theorems [1] and [21 Chernoff's formula and Hoeffding's formula in simple 
hypothesis testing are essential, and are summarized as follows. Let the probability p on fi 
be the null hypothesis and q be the alternative hypothesis. When we discuss a hypothesis 
testing problem concerning n-i.i.d. data, we call a sequence A = {An} a test, where An is 
an acceptance region, which is a subset of fi". The first error probability ei{An) and the 
second error probability 62 (A„) are defined as 

eMn) ■■= 1 - p"{An), e2{An) 1= g"(A„), 

and their exponents are given by 

el{A) :=liminf — logei(A„) 

n^oo n 

62 (A) := liminf loge2(A„). 

n — >oo Ti 
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ChernofF (jChernoff (1952)1 ) evaluated the exponent of the sum of the two errors as 



sup lim — log(ei(l„) + e2(i'„)) =supmin{e*(i'),e;(i*)}= sup P{p\\q), (20) 

^ n^oo n ^ 0<s<l 

which is essential for our proof of Theorem [TJ This bound is achieved by both of the 

hkelihood tests {w„ e f7"|p"(w„) > g"(cJ„)} and {cJ„ G ri"|p"(w„) > q^(u}„)}- 

HoefFding proved another formula for simple hypothesis testing (JHoeffding (1965)1 ): 



sup 62 (A) = sup ■ 

A:el{A)>r 0<s<l 



-sr + /"(pIIq) 



(21) 



This formula is essential for our proof of Theorem [2l 

Proof of Theorem]^ Applying equation (|20p to the two hypotheses pe~e and pe+e, we 
obtain 

inf p{f,e',e)<imn{l3+{f,e-e,e),l3-{f,e + e,e)}< sup r{pg.,\\pe+,). 
e-e<e'<e+t o<s<i 



Taking the limit e — > 0, we have 

1 



lim — - sup inf P{T,e' ,e) < 2" sup /, 



e^+O g[e) f. 



T:WC 



-t<8'<e+t 



0<s<l 



g.e- 



(22) 



In the regular case, this method was used by Sievers (jSievers (1978)1 ) 
Proof of Theorem O' Hoeffding's formula ([2T|l yields 

inf I3{f,e',{l-7j)e) 

e-{l-ri)e<e'<e + {l-r,)e 



<p-{T,9+-e,{l~rj)e) 



< sup ■ 

0<s<l 



< sup 

0<s<l 



-sf3+{f,e - ^e,Tje) + r{pg_.J\pg^,J 

1-s 
-s ini0-,,e<e'<e+7,€ P{T, 0' , i]e) + I'ipg_i,\\pg+i,) 



1-s 



The uniformity of P^ guarantees that 



a2{T,e){l-r^r< sup ■ 

0<s<l 



-sa2{T,9)rj'' + Ilg 
1-s 



From (l23l). we have 



a2{T,6) < sup < a 



a(7y«,(l-r/)'')eUa;,y) 



y < max 



-sx + If, 



0<s<l 1 — S 



We define the set Ci and ao as 



Ci := < {x,v) 



^^ + ^le 



V > sup — 

o<t<i i — I 



ao ■.^sup{a\{aT]'',a{l-T])'^) ^ int(Ci), < Vr; < 1}. 



(23) 



x,y>0} , < V?7 < 1 

(24) 



(25) 
(26) 
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Note that the function s i-^ I^ g is concave. We define the convex function g{x) and another 
set C2 as 

9{x):=i^"{^-{^)^) 0^^^"o (27) 

[ X > ao 

C,:^{{x,y)\y>g{x)}. (28) 

Since inequality ([24|) guarantees 

(a2(T, e)?f,a2if, 0){l - TlT) i int(Ci), < Vt? < 1, 

the inequality 

a2{f,e)<ao (29) 

holds. Relations ([26|l . ([27]) . and ([28| guarantee the relation 

Ci C C2. (30) 

Applying Lemma [7l we have 

/ -^2; + /* \ 
inf (sx + (1 — ,s)w) = inf sx + (1 — s) sup ^^- 

(x,v)<^ci^ '■ ' x>oy o<t<i 1-t y 

= inf sup ^ ' \ >^'' = Ilg. (31) 

In the following, we divide our situation into three cases k >1,k=1,1 > k >0. When 
K > 1, relation ([50)1 guarantees that 

aos(l-s) fs^ + (1-3)^) ^ao min (577'" + (1 - s)(l - 77)'^) 
V / <)) <i 

= inf (sa; + (1 - s)y) < inf (sx + (1 - s)y) = /^ (,. 
(i,a)eC2 (2:,a)eCi "' 

Therefore, 

(s^ + (l-s)^)'"' 

a2(r,0) < ao < ^ 7. ^ lie, 

s(l — s) "' 

which implies (|15p . 

When K ~ I, similarly, we can easily prove 

Finally, we consider the case where k < 1. Since function g is concave on (0, ao), there 
exists 770 G (0, 1) such that 

ao(%",(i-%)")eCinC|. (32) 
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Since Ci and C2 are convex, there exists a real number sq S (0,1) such that 

sup {sqx + {1 - so)y) = ao{soT]o + {1 - so){l - Tjo)") ^ inf {sqx + {1 - so)y) . 

In general, for any s G (0, 1), using (|31[) . we obtain 

aos{l-s)(s^ + {l-s)^) =aa sup (577'* + (1 - s)(l - 77)'^) 

^ ^ 0<?7<1 

= sup (sx + {1 - s)y) > ao {stjS + (1 - s){l - r]oT) > M {sx + (1 - s)y) = Tg, 

{x,y)GC^ (2:,i;)eCi 

which lead to (fT5|) . ■ 

Proof of Theorem\^ It is trivial in the case of k = 1. When k > 1, it is also trivial 
because 

Igg /I 1 \K-i 



0<s<l s(l-s) V ' y 9,« p^^^^ 9.f 

Next, we consider the case k < 1. The inequality 

sup -JE:L_(s^ + (l-s)^y <2^ sup Ilg (33) 

0<s<l S(l - S) \ / 0<s<l 

follows from the two inequalities 

Ilg< sup lie (34) 

0<s<l 

-^(.^ + (l-.)^)''-^2^ (35) 

We thus obtain ([T7]). In the following, we prove that the equality of (|17p implies P^ and 
(fT5)) implies P^ and the equality of (fT7|) in the case where k < 1. If we assume that the 
equality of p3)) holds, the equalities of ((34)) and p5|) hold at the same s. The equality of 
((55|) holds if and only if s = i. Therefore, 

^g,9 = sup Ig^g, 
0<s<l 

which is equivalent to (fT5)). If we assume that pS]) holds, inequality ([55)1 guarantees that 

S2(0) < 2''/|,. (36) 

Substituting ^ into s at the left hand side (LHS) in the definition of 7x2 {&) , we obtain 

MO) > 2^ lie- 
Thus, equation (fT9|) holds. Combining (fTS]) and (fTg]) . we obtain the equahty of (fTT]) . ■ 



Two non-regular extensions of large deviation bound 9 
5. Exponential rates of useful estimators and their slopes 

In the following, we discuss the exponential rate (3{T, 6, e) of a useful estimator T for a 
location shift family {f{x — d)\d ^ M.}, where / is a probability density function (pdf) on 
M. In particular, we focus on the case where the support of / is (a, b). Further, we assume 
that the pdf / is C^ continuous and satisfies that 



f{^)- 


= A,{x- 


-ar^-\ 


X ^ a + 


/(^) 


= A2{b- 


-x)''^-\ 


x-^b-0, 


l-ld-f-Q 


rlic'-f-yi Km 


fir^nf ^ 


■\i-.f-' 



When its support is a half line (0, oo) as for the gamma distribution and WeibuU dis- 
tribution, our situation results in the above case where A2 = if / is C^ continuous and 
limji^oo \'^^ogf{x)\ < 00. Also, when ki > K2, our situation results in the above case 
where A2 — 0. 

In the first step, we will treat estimators useful for point estimation. After this discus- 
sion, we will discuss estimators useful for interval estimation. 



5. 1 . Exponential rates of estimators useful for point estimation 

When the support of / is (a, 6), the two estimators 6'„ :— max{a;i, . . . , x„} — b and 0„ := 
min{a;i, . . . ,a;„} — a are useful for point estimation. These performances are characterized 
as follows. 

Lemma 1 . The estimators 6 and 6 satisfy 

/3+{l9,e) ^ -log I J^ /(x)dxj, /3-(|,0,e) = oo (37) 

P+ite,e)^^, p-ito,e) = ~\oglj^ fix)dxy (38) 

Proof: Since 

(h— \ " 

J^ V(x)dxj , f^{0„<0} = O 

/ b \ " 

fnOn <9^e}=IJ^ fix) dx J , /,"{&„ > 0} = 0, 



we obtain ^ and dSH]). ■ 

In order to strike a balance between two exponential rates (3'^{T,9,e) and fS^iTjOje), we 
define the convex combination (CC) estimator 9cc,x '■— {9cc,\.n '■= A^„ -I- (1 — A)6'„} with 

the ratio A : 1 — A of the two estimators 9 and 9, where < A < 1. These are characterized 
as follows. 
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Lemma 2. The convex combination (CC) estimator 6cc.\ satisfies that 



/?+(0cc,A,e,e) = -log( / f{x)dx\ (39) 

rb \ 

p-{9ccA,e,e) ^~\og\ I f{x)dx]. (40) 



Proof: Define w„ := maxlwi, . . . , w„}, w„ :— niinjwi, . . . , w„}. Since the estimators 

9_, 9, and 9cc,\ are covariant for location shift, we may discuss only the case that 6* = 0. 
From the relation 9^>9> 9—e, we obtain the second equation of ([ST]) . Its joint probability 
density function /n(wn,w„) is given by 



By defining 



fnPn,^n) := <( ' ^ ' V^^" ' ' ' J ' '-''" ' "' _ ^ "" (41) 

UJn < UJ„ . 



5(w„,a;„) := < — 

[0 UJn <Ul^, 

The equation (|4T|) yields 

Pe{9cc,x.n<9-e)= f _ n(n - l).g(c^„,cJ„)"/(ct;J/(ZJ„) d^„dcJ„. (42) 

From the continuity of f{uj_^), f(uJn) and (7(a;„,(U„), the equations 

lim -logpg(6'cc.A,n < ^-e) = 1™ - sup logg(w„,ZU„) 



= - log I / f{x) dx 



hold. This implies the first equation of (|37p and ([55]) . In addition, we can similarly show 
the same fact for (|40|) . ■ 

Next, we focus on the maximum likelihood estimator 9ml '■= {9ML,n}- 

Lemma 3. When the function x i-^ log /(a;) is concave, MLE 9ml satisfies that 

P+{9ML,9,e) = sup -log / exp ( ^tQ^^^) f{x)dx (43) 

= sup -log/ e^^{~t^-^]f{x + t)dx (44) 

t>0 J a \ fix) J 



p-{9ML,9,e) = sup-log f exp ft^^^^) /(a;) da; (45) 



sup — log 

t>0 
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Also, we have the following evaluations in the same assumption. 

f3+{0ML,O,e)> snp P{fe\\fe+e) (47) 

0<s<l 

nOML,d,e)> sup r(/e_,||/e). (48) 

0<s<l 

This lemnia is essentially proved as a one step of the proof of the main theorem in the paper 
(|Fu (1973)1 ). However, he proved the main theorem with a more general assumption and 
a different notation. Hence, it is not easy to find the relationship between the notation of 
the present paper and that of Fu's paper. Further, since his proof of this part is too short, 
it seems that a non-expert of large deviation theory cannot follow his proof. Therefore, for 
the reader's convenience, a proof of this lemma is given as follows. 

Proof: Equations (|44|) and (|46p are trivial. We prove (|43|l . From the assumption that 
for any af„ :— (xi, . . . , a;„), the function 9 i-^ J2^=i ^ogf{xi — 6) is concave on {9{xn),Q_{xn)), 
the function 9 i-^- Y^^^i^^i^i) is monotonically decreasing, where le{x) := — f(^Zg) on 

{9{xn),9{xn))- When Xn belongs to the support of /g', the condition 9ML,n{xn) > 9' is 
equivalent to the condition 






{x^) > 0. 

z—l 

Denoting the conditional probability f{A\x G B} under the condition x G i?, we can 
evaluate 

\im--\ogrg'{9ML,n>0 + e} 
n 

= \m-i~-\ogf^'{9ML.n>0 + e\x,,e{a + e,b + er}--logf^\a + e,b + er 
n n 

= hm -1 log/;,, J - V le+M) > I - log / f{x) dx, (49) 



- log/;, J - E W(x.) > I - log / 



where the probability density function /e,, is defined on the support (a + e, b) by 

ta+, f{x) dx 

Chernoff's theorem (Theorem 3.1 in Bahadur (JBahadur (1971)1 )) guarantees that 

1 1 " 

hm — log/^^,{-^/e+e(x.)>0} 
n n ^-^ 

z— 1 

= sup - log / exp {tlg+^{x)) fe,e{x) dx 

t>0 Ja+e 

= sup - log / exp f-tQ^-^] fix) dx + log / fix) dx. (50) 

t>0 Ja+e V J{X -9) J Ja+e 

Combining (P5|) and (|50p . we obtain (|15)l . Similarly, we can prove (|45p. 
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Next, in order to show ([T7|) . we choose a real number S > 0. When 

fe'-si^n) < f^-ixn), (51) 

the condition 

OMLA^n) > 0' (52) 

holds. Hence, substituting 9 — e and e into 0' and ^, respectively, we have 

fnOML.ni^n) > + e} < f^fe i^n) < f6+e{Xn)}. (53) 

As is mentioned at (|20|) in SectionlH the exponential rate of the RHS is equal to supo<s<i I'^ife Wfe+a)- 
Thus, we obtain (j47p . Similarly, we can prove 



Lemma 4. When f{x) is monotonically decreasing, the MLE 0ML,n equals the estimator 



Proof: For any data a;„ := (xi, . . . , a;„), ii < ^„(a?n)), f{xi — 0) . . . f(xn — 0) = 0- 
Conversely, if 6* > ^„(x„)), the obtained f{xi -0) < f{xi- 0n{xn))- Thus, 0„ is the MLE. 



5.2. Slopes of exponential rates of estimators useful for point estimation 

We discuss the slopes of exponential rates of estimators discussed in the above. First, we 
focus on the estimators d_ and 0. From Lemma [l] the estimators d_ and satisfy that 

/3(l,e,e) = — e"^+o(e"0, /?(f, 0, e) = — e"^ + o(e"=). (54) 

Next, we proceed to the convex combination estimator 0cc,n- When ki = K2 = k, 
Lemma [2] yields the equations 

/?+(0cc,A,e,e) = ^e'^ + o(e«), riOcc,x,d,e) - .f' ^. e^ + o^e^- 
kX k(1 — A) 

j_ 
Since Aq :— ^ ^ i — ^^W^^^o<\<i min{^, ^,-^^^, }, the relations 



1 1 



(55) 



K 

hold. 

When the function x i-^ log /(a;) is concave, the relations (|47)) and (|48)) yield that 

a{0ML,0)>^^. (56) 

Further, when f{x) is monotonically decreasing. Lemma |4] and (j54p imply that 

/3(^lfL,0,e)^— e"i+o(e''i). (57) 
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5.3. Exponential rates of estimators for interval estimations 

In order to improve the exponential rate for a fixed width 2e, we focus on the Ukehhood 
ratio estimator, which is discussed by Huber, Sievers, and Fu from the large deviation 
viewpoint concerning the regular familv lJHuber (1968)USievers (^1978)tlFu (1985)1 ). Assume 
that the function d^{x) :— f{x + e)//(x — e) is monotonically decreasing w.r.t. x when 
both f[x + e) and f{x — e) are not zero. Then, we can define the likelihood ratio estimator 
(^LR,e '■= {(^LR.e,n{xi, ■ ■ ■ , Xn)}, which depends on the constant e > 0, as shown by 

OLR,e,n := ^ (sup{z|fc(z) < 0} + inf{z|fc(z) > 0}) , (58) 

where the monotonically decreasing function k{z) is defined by 

1 " 
k{z) := - V (log /(x. - z + e) - log /(x, - z - e)) . (59) 

n ^ — ' 

Note that when log/(x) is concave, the above condition is satisfied. This definition is well 
defined although the monotonically decreasing function /c(z) is not continuous. 

If the support of / is (a, 6), we need to modify the definition as follows. In this case, we 
modify the estimator BhR.t.n by using the two estimators On and Q_^. When 0^^ — On > 2e, 
the estimated value is defined by (|58p in the interval (£„ — e, 0„ + e). When ^„ — On < 2e, 
we define 0i,_R,e,n := 5 (^„ + On)- Moreover, when the support of / is a half line (0, 00), the 
estimated value is defined by ([55]) in the half line {0^ — e, cxd). 

Then, the exponentia l rate is charac terized as follows. (A reg ular versi on of this l emma 
was discussed by Huber (JHuber (1968)L Sievers (jSievers (1978)1 ). and Fu (JFu (1985)1 ). 1 

Lemma 5. When log f{x) is concave, the equation 

min{l3-{0LR,e,O,e),l3+i0LR,e,O,e)}^ sup Pife^e\\fe+e)- (60) 

0<s<l 

holds, where fe{x) :— f{x — 0). 

Therefore, the equality of inequality ITTI) holds in this case. 

Proof: Notethatf3+{0LR,,,0,e)^l3+{OLR,e,0-e,e)and(3~{OLR,e,O,e)=f]-{OLR,e,0+ 
e, e) because of the shift-invariance. From the concavity, the condition 0LR.e.n < is equiv- 
alent to the condition sup{z|/c(z) < 0} < 0, which implies that k{0) > 0. Thus, we have 

J2 log/(a:^ -0 + e)- log/(a;, - ^ - e) > 0. 

i 

Conversely, the condition 

Y,^ogf{x,-O + e)-logf{x,-0-e) >0 

i 

implies that k{0) > 0. Thus, we have sup{z|fc(z) < 0} < 0, which is equivalent to the 
condition OLR.c.n < 0. Therefore, we have the relations 

{fe^ei^n) > f^+^Xn)} C {OLR^,,n < 0} C {/;_,(f„) > f^+A^n)}- 



1 4 Masahito Hayashi 
Similarly, we can prove 

{/fl\,(x„) > /^,(a^„)} C {0Li,,.,„ >e}c {/,\,(f„) > /r,(x„)}. 
Applying (PO]) . we obtain 

niin{/3-(0^M,e,0 + e,e),/3+(0M,.,^-e,e)}= sup r(/e-.l|/e+.), 

0<s<l 

which implies equation (|60p . ■ 

Next, in order to improve the estimator for a fixed value e, we define the estimator 
9^ := {6^ „ := £„ — e}. This estimator satisfies the following lemma. 

Lemma 6. The estimator 9^ satisfies 

P+{l,9,e) = -\oglj^ f{x)dxy (61) 

/3-(i,^,e) = oo. (62) 

Further, when f{x) is monotonically decreasing, the exponential rate of the estimator 9^ has 
another form 

P+{l,9,e)= sup r{fg.,\\fg+,). (63) 

0<s<l 

Proof: From the construction of ^^, the relations ([6T|) and ((62|) follow from ([37|l . Then, 
we proceed to the proof of ([55)1 . Since ^„ > 0, we have ^^ „ > 9 ~ e, which implies ([55)1 . If 
^e,n > 6* + e, we have ^„ > 6* + 2e. Thus, /(a;, -{9 + 2e))> f{x^ - 9) for any i = 1, . . . ,n. 
Therefore, 

Je+2t{^n) ^ fe [Xn}- 

Conversely, if fg^2ei^n) > fgi^n), we have ^„ > + 2e. Thus, 

/e"f2.{^n|/^V2.(^") < /e"(2^„)} - /^V2.{2^«|^„ < + 26} = 0. 

Since the likelihood test {xn|^„ > 6* + 2e} achieves the optimal rate (|20p . we have 

lim log f'g'{Xn I /^V2e (^" ) > fg (Xn ) } 

n 

= lim-l log (/,"{x„|/,V(f„) > /,"(X„)} + /;+2e{2'„|/e"+2e(^rO < /e"(a?„)}) 

= sup I%fg\\fg+2e)^ sup /"(/e_e||/e+e). 
0<s<l 0<s<l 



5.4. Slopes of exponential rates of estimators useful for interval estimation 

Next, we proceed to the slopes of exponential rates. Concerning the estimator 9^, from ((6T|) 
and ([62|). we obtain the following characterization: 

Pil,9,e)^Ai—e'''+oie^'). (64) 
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Further, when f{x) is monotonically decreasing, the equation (j63p implies the equation 

hrn/%M.S,(,). (65) 

When the function log /(a;) is concave, Lemma [5] yields the equation 

m -— 

*+o 5(e) 



j.^ m,K^^-(^)^ (66) 



Thus, the following theorem holds. 

Theorem 4. When the function log/(x) is concave or monotonically decreasing, the 
relation 

holds. 

Therefore, if the condition of the above theorem holds and condition p^ in Theorem [3] 
does not hold, i.e., H g is not symmetric, then the first criterion is different from the second 
one, i.e., 

ai{9)>a2{e). (67) 

6. Conclusion 

We have discussed large deviation theories under a more general setting than existing stud- 
ies. For this purpose, we introduced two criteria for the bound of estimation accuracy from 
the large deviation viewpoint. One criterion ai{9) corresponds to the interval estimation 
with taking the limit that the width of error bar goes to 0. The other a2{0) corresponds to 
the point estimation. The upper bounds of them are given by the limit of the relative Renyi 
entropy. These characterizations have been derived by the method of simple hypothesis 
testing. We have also calculated the slope of the exponential decreasing rates of several 
estimators. As a result, we have succeeded in deriving upper bounds ai{9) and a2{d) of 
ai{9) and ol2{9) and a necessary and sufficient condition for the coincidence of these two 
upper bounds ai{9) and a.2(9). In the next step, we have treated several estimators as 
candidates to attain the optimal bounds ai{9) and a2{9) in a local shift family. That is, 
we derived lower bounds of ai{9) and ot2{9). Further, we derived a sufficient condition for 
the gap between two criteria ai{9) and a2{9). 

Unfortunately, we cannot calculate two criteria ai{9) and a2{9) in the concrete exam- 
ples. For this purpose, we need t o calculate the li mit of the relative Renyi entropy, which 
was discussed in another paper (JHavashi (2002)bl ). In the next p aper, we will trea t this 



calculation based on the obtained result and the result by Hayashi (JHavashi ('2002') bl ). 

Historically, N agaoka initiated a discussion of tw o kinds of large deviation bounds in a 
quantu m setting (INagaoka (^199411 : iNagaoka (^199211 ). and Hayashi discussed these in more 



depth (JHavashi (2002)al ). The two kinds of large deviation bounds do not necessarily co- 



incide in a quantum setting. In quantum setting, this difference corresponds to the non- 
uniqueness of quantum extension of Fisher information. In the quantum case, ai{9) cor- 
responds to Kubo-Mori-Bogoljubov (KMB) inner product and a2{9) does to Symmetric 
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logarithmic derivative (SLD) inner product. This research is strongly motivated by this 
quantum study. In the quantum case, the family of estimators attaining the bound Si {9) 
depends on the true parameter. However, in some of non-regular location families, such 
a family of estimators does not depends on the true parameter. This is different point 
between our setting and quantum setting. Gaining an understanding of these differences 
from a unified viewpoint remains a goal for the future. 
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A. Concave function 

Lemma 7. When a concave function f > is defined in (0, 1), 

■ f ^n \ -to + f{t) {s - t)x + (1 - s)f{t) 

inisx + (1 - s) sup = mf sup = /(s). 

2;>0 0<t<l l — t 2;>Oo<t<i i — t 

Proof: Substituting s into t we have 

/(.) < sup (--^)-+(l--)/W. 
o<t<i 1 — i 

Taking the infimum infj-^o? we obtain 

.. ^ . . f (s-t)x+(l-3)f(t) 

f{s) < mf sup . 

^>"o<t<i t - i 

Next, we proceed to the opposite inequality. From the concavity of /, we can define the 
upper derivative /' and the lower derivative /' as 

eto+0 e — £to+o e 

Since the concavity guarantees that 

^^(i±il-M<;'(,)<77(,)</(fW(i^, Ve>0, 
e — e 

we obtain 

(s - s)if{.s) + (1 ~ s)T{s)) + (1 - s)f{.s) is - t){f{s) + (1 - s)r{.s)) + (1 - .s)f{t) 
1-s 1-t 

=1~T (/G^) - fit) + it- '^)T{s)) > 0, V,s, Vt G (0, 1). 
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Therefore, 

..^> {s - mis) + ji -s)r{s)) + {I -s) fit) 

0<i<l i — t 

>.^f g^p is-t)ifi.s) + il-.s)Ti.s)) + il-s)fit) _ 
~ x>Oo<:t<i 1 - t 

The proof is now complete. ■ 

B. Other lemmas 



Lemma 8. When g is strictly monotonically decreasing and continuous, giO) — 0. and 

g(xf) 



the limit lim^^+o T-! exists, there exists k > such that 



a;" = a; > 0. (68) 

Proof: Let hix) be the RHS of ^. Since 

lini ^ . hm ^ lim ^, 
e-+0 c,(e) 6^+0 giye) £^+0 g(e) 

hixy) = hix)hiy). Thus, there exists k > satisfying ([68]) . ■ 



Lemma 9. For < s < ^, the inequalities 

2sliip\\q)<rip\\q)<2il-s)I^p\\q) 



hold. 



Proof: Since (^) + I j-rjj ) = 1; the Holder inequality guarantees that 



fioj)q'-^ioj)doj= ip%uj)q^oj)) iq'-^%uj)) duj 
n Jn 

< (^^ ip-^ii,)q%u;))^ du^ ■ (^^^ (gl-2-(c.))^ dc^) 
= I / p^iu!)q^iu!) du! 



Thus, we obtain 

rip\\q)>2sl^p\\q). 
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-1 



Similarly, since (2(1 — ,s)) + ( [_2l ) = 1, we can apply the Holder inequality as 

p^ {uj)q^ (uj) duj 
n 

{p%uj)q^-''{uj)) ^^ p"^^ (uj) duo 

1 1-23 

f r , , 1 2fi ji \ 2(1-=) / r 1-23 2(1-3) \ 2(1-3) 

( / {p'{oj)q^-'{Lj))'-''-^^ ^ ' dLj\ • ( / p^i^^-^^^{uj)duj\ 



in 
< 



, 2(1-3) 

p^ {uj)q ^ (bj) dio 



in 
which implies that 



iHph)>i^^^f{ph)- 
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